A zero-shot reinforcement learning strategy for autonomous guidewire navigation.

PURPOSE: The treatment of cardiovascular diseases requires complex and challenging navigation of a guidewire and catheter. This often leads to lengthy interventions during which the patient and clinician are exposed to X-ray radiation. Deep reinforcement learning approaches have shown promise in learning this task and may be the key to automating catheter navigation during robotized interventions. Yet, existing training methods show limited capabilities at generalizing to unseen vascular anatomies, requiring to be retrained each time the geometry changes. METHODS: In this paper, we propose a zero-shot learning strategy for three-dimensional autonomous endovascular navigation. Using a very small training set of branching patterns, our reinforcement learning algorithm is able to learn a control that can then be applied to unseen vascular anatomies without retraining. RESULTS: We demonstrate our method on 4 different vascular systems, with an average success rate of 95% at reaching random targets on these anatomies. Our strategy is also computationally efficient, allowing the training of our controller to be performed in only 2 h. CONCLUSION: Our training method proved its ability to navigate unseen geometries with different characteristics, thanks to a nearly shape-invariant observation space.

Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System.

INTRODUCTION: Intraoperative navigation during liver resection remains difficult and requires high radiologic skills because liver anatomy is complex and patient-specific. Augmented reality (AR) during open liver surgery could be helpful to guide hepatectomies and optimize resection margins but faces many challenges when large parenchymal deformations take place. We aimed to experiment a new vision-based AR to assess its clinical feasibility and anatomical accuracy. PATIENTS AND METHODS: Based on preoperative CT scan 3-D segmentations, we applied a non-rigid registration method, integrating a physics-based elastic model of the liver, computed in real time using an efficient finite element method. To fit the actual deformations, the model was driven by data provided by a single RGB-D camera. Five livers were considered in this experiment. In vivo AR was performed during hepatectomy (n = 4), with manual handling of the livers resulting in large realistic deformations. Ex vivo experiment (n = 1) consisted in repeated CT scans of explanted whole organ carrying internal metallic landmarks, in fixed deformations, and allowed us to analyze our estimated deformations and quantify spatial errors. RESULTS: In vivo AR tests were successfully achieved in all patients with a fast and agile setup installation (< 10 min) and real-time overlay of the virtual anatomy onto the surgical field displayed on an external screen. In addition, an ex vivo quantification demonstrated a 7.9 mm root mean square error for the registration of internal landmarks. CONCLUSION: These first experiments of a markerless AR provided promising results, requiring very little equipment and setup time, yet providing real-time AR with satisfactory 3D accuracy. These results must be confirmed in a larger prospective study to definitively assess the impact of such minimally invasive technology on pathological margins and oncological outcomes.

Estimation of boundary conditions for patient-specific liver simulation during augmented surgery.

PURPOSE: Augmented reality can improve the outcome of hepatic surgeries, assuming an accurate liver model is available to estimate the position of internal structures. While researchers have proposed patient-specific liver simulations, very few have addressed the question of boundary conditions. Resulting mainly from ligaments attached to the liver, they are not visible in preoperative images, yet play a key role in the computation of the deformation. METHOD: We propose to estimate both the location and stiffness of ligaments by using a combination of a statistical atlas, numerical simulation, and Bayesian inference. Ligaments are modeled as polynomial springs connected to a liver finite element model. They are initialized using an anatomical atlas and stiffness properties taken from the literature. These characteristics are then corrected using a reduced-order unscented Kalman filter based on observations taken from the laparoscopic image stream. RESULTS: Our approach is evaluated using synthetic data and phantom data. By relying on a simplified representation of the ligaments to speed up computation times, it is not estimating the true characteristics of ligaments. However, results show that our estimation of the boundary conditions still improves the accuracy of the simulation by 75% when compared to typical methods involving Dirichlet boundary conditions. CONCLUSION: By estimating patient-specific boundary conditions, using tracked liver motion from RGB-D data, our approach significantly improves the accuracy of the liver model. The method inherently handles noisy observations, a substantial feature in the context of augmented reality.

Constrained stochastic state estimation of deformable 1D objects: Application to single-view 3D reconstruction of catheters with radio-opaque markers.

Minimally invasive fluoroscopy-based procedures are the gold standard for diagnosis and treatment of various pathologies of the cardiovascular system. This kind of procedures imply for the clinicians to infer the 3D shape of the device from 2D images, which is known to be an ill-posed problem. In this paper we present a method to reconstruct the 3D shape of the interventional device, with the aim of improving the navigation. The method combines a physics-based simulation with non-linear Bayesian filter. Whereas the physics-based model provides a prediction of the shape of the device navigating within the blood vessels (taking into account non-linear interactions between the catheter and the surrounding anatomy), an Unscented Kalman Filter is used to correct the navigation model using 2D image features as external observations. The proposed framework has been evaluated on both synthetic and real data, under different model parameterizations, filter parameters tuning and external observations data-sets. Comparing the reconstructed 3D shape with a known ground truth, for the synthetic data-set, we obtained average values for 3D Hausdorff Distance of 0.81±0.53mm, for the 3D mean distance at the segment of 0.37±0.17mm and an average 3D tip error of 0.24±0.13mm. For the real data-set,we obtained an average 3D Hausdorff distance of 1.74±0.77mm, a average 3D mean distance at the distal segment of 0.91±0.14mm, an average 3D error on the tip of 0.53±0.09mm. These results show the ability of our method to retrieve the 3D shape of the device, under a variety of filter parameterizations and challenging conditions: uncertainties on model parameterization, ambiguous views and non-linear complex phenomena such as stick and slip motions.

Deformation Aware Augmented Reality for Craniotomy using 3D/2D Non-rigid Registration of Cortical Vessels.

Intra-operative brain shift is a well-known phenomenon that describes non-rigid deformation of brain tissues due to gravity and loss of cerebrospinal fluid among other phenomena. This has a negative influence on surgical outcome that is often based on pre-operative planning where the brain shift is not considered. We present a novel brain-shift aware Augmented Reality method to align pre-operative 3D data onto the deformed brain surface viewed through a surgical microscope. We formulate our non-rigid registration as a Shape-from-Template problem. A pre-operative 3D wire-like deformable model is registered onto a single 2D image of the cortical vessels, which is automatically segmented. This 3D/2D registration drives the underlying brain structures, such as tumors, and compensates for the brain shift in sub-cortical regions. We evaluated our approach on simulated and real data composed of 6 patients. It achieved good quantitative and qualitative results making it suitable for neurosurgical guidance.

Alignment of Cortical Vessels viewed through the Surgical Microscope with Preoperative Imaging to Compensate for Brain Shift.

Brain shift is a non-rigid deformation of brain tissue that is affected by loss of cerebrospinal fluid, tissue manipulation and gravity among other phenomena. This deformation can negatively influence the outcome of a surgical procedure since surgical planning based on pre-operative image becomes less valid. We present a novel method to compensate for brain shift that maps preoperative image data to the deformed brain during intra-operative neurosurgical procedures and thus increases the likelihood of achieving a gross total resection while decreasing the risk to healthy tissue surrounding the tumor. Through a 3D/2D non-rigid registration process, a 3D articulated model derived from pre-operative imaging is aligned onto 2D images of the vessels viewed through the surgical miscroscopic intra-operatively. The articulated 3D vessels constrain a volumetric biomechanical model of the brain to propagate cortical vessel deformation to the parenchyma and in turn to the tumor. The 3D/2D non-rigid registration is performed using an energy minimization approach that satisfies both projective and physical constraints. Our method is evaluated on real and synthetic data of human brain showing both quantitative and qualitative results and exhibiting its particular suitability for real-time surgical guidance.

Definitions of Computer-Assisted Surgery and Intervention, Image-Guided Surgery and Intervention, Hybrid Operating Room, and Guidance Systems: Strasbourg International Consensus Study.

OBJECTIVE: To develop consensus definitions of image-guided surgery, computer-assisted surgery, hybrid operating room, and surgical navigation systems. SUMMARY BACKGROUND DATA: The use of minimally invasive procedures has increased tremendously over the past 2 decades, but terminology related to image-guided minimally invasive procedures has not been standardized, which is a barrier to clear communication. METHODS: Experts in image-guided techniques and specialized engineers were invited to engage in a systematic process to develop consensus definitions of the key terms listed above. The process was designed following review of common consensus-development methodologies and included participation in 4 online surveys and a post-surveys face-to-face panel meeting held in Strasbourg, France. RESULTS: The experts settled on the terms computer-assisted surgery and intervention, image-guided surgery and intervention, hybrid operating room, and guidance systems and agreed-upon definitions of these terms, with rates of consensus of more than 80% for each term. The methodology used proved to be a compelling strategy to overcome the current difficulties related to data growth rates and technological convergence in this field. CONCLUSIONS: Our multidisciplinary collaborative approach resulted in consensus definitions that may improve communication, knowledge transfer, collaboration, and research in the rapidly changing field of image-guided minimally invasive techniques.

Elastic Registration Based on Compliance Analysis and Biomechanical Graph Matching.

An automatic elastic registration method suited for vascularized organs is proposed. The vasculature in both the preoperative and intra-operative images is represented as a graph. A typical application of this method is the fusion of pre-operative information onto the organ during surgery, to compensate for the limited details provided by the intra-operative imaging modality (e.g. cone beam CT) and to cope with changes in the shape of the organ. Due to image modalities differences and organ deformation, each graph has a different topology and shape. The adaptive compliance graph matching (ACGM) method presented does not require any manual initialization, handles intra-operative nonrigid deformations of up to 65 mm and computes a complete displacement field over the organ from only the matched vasculature. ACGM is better than the previous biomechanical graph matching method (Garcia Guevara et al. IJCARS, 2018) (BGM) because it uses an efficient biomechanical vascularized liver model to compute the organ's transformation and the vessels bifurcations compliance. This allows to efficiently find the best graph matches with a novel compliance-based adaptive search. These contributions are evaluated on 10 realistic synthetic and 2 porcine automatically segmented datasets. ACGM obtains better target registration error (TRE) than BGM, with an average TRE in the real datasets of 4.2 mm compared to 6.5 mm, respectively. It also is up to one order of magnitude faster, less dependent on the parameters used and more robust to noise.

Simulation of hyperelastic materials in real-time using deep learning.

The finite element method (FEM) is among the most commonly used numerical methods for solving engineering problems. Due to its computational cost, various ideas have been introduced to reduce computation times, such as domain decomposition, parallel computing, adaptive meshing, and model order reduction. In this paper we present U-Mesh: A data-driven method based on a U-Net architecture that approximates the non-linear relation between a contact force and the displacement field computed by a FEM algorithm. We show that deep learning, one of the latest machine learning methods based on artificial neural networks, can enhance computational mechanics through its ability to encode highly non-linear models in a compact form. Our method is applied to three benchmark examples: a cantilever beam, an L-shape and a liver model subject to moving punctual loads. A comparison between our method and proper orthogonal decomposition (POD) is done through the paper. The results show that U-Mesh can perform very fast simulations on various geometries and topologies, mesh resolutions and number of input forces with very small errors.

Force classification during robotic interventions through simulation-trained neural networks.

PURPOSE: Intravitreal injection is among the most frequent treatment strategies for chronic ophthalmic diseases. The last decade has seen a serious increase in the number of intravitreal injections, and with it, adverse effects and drawbacks. To tackle these problems, medical assistive devices for robotized injections have been suggested and are projected to enhance delivery mechanisms for a new generation of pharmacological solutions. In this paper, we present a method aimed at improving the safety characteristics of upcoming robotic systems. Our vision-based method uses a combination of 2D OCT data, numerical simulation and machine learning to classify the range of the force applied by an injection needle on the sclera. METHODS: We design a neural network to classify force ranges from optical coherence tomography (OCT) images of the sclera directly. To avoid the need for large real data sets, the network is trained on images of simulated deformed sclera. This simulation is based on a finite element method, and the model is parameterized using a Bayesian filter applied to observations of the deformation in OCT images. RESULTS: We validate our approach on real OCT data collected on five ex vivo porcine eyes using a robotically guided needle. The thorough parameterization of the simulations leads to a very good agreement between the virtually generated samples used to train the network and the real OCT acquisitions. Results show that the applied force range on real data can be predicted with 93% accuracy. CONCLUSIONS: Through a simulation-trained neural network, our approach estimates the force range applied by a robotically guided needle on the sclera based solely on a single OCT slice of the deformed sclera. Being real-time, this solution can be integrated in the control loop of the system, permitting the prompt withdrawal of the needle for safety reasons.

An open electromagnetic tracking framework applied to targeted liver tumour ablation.

PURPOSE: Electromagnetic tracking is a core platform technology in the navigation and visualisation of image-guided procedures. The technology provides high tracking accuracy in non-line-of-sight environments, allowing instrument navigation in locations where optical tracking is not feasible. EMT can be beneficial in applications such as percutaneous radiofrequency ablation for the treatment of hepatic lesions where the needle tip may be obscured due to difficult liver environments (e.g subcutaneous fat or ablation artefacts). Advances in the field of EMT include novel methods of improving tracking system accuracy, precision and error compensation capabilities, though such system-level improvements cannot be readily incorporated in current therapy applications due to the 'blackbox' nature of commercial tracking solving algorithms. METHODS: This paper defines a software framework to allow novel EMT designs, and improvements become part of the global design process for image-guided interventions. An exemplary framework is implemented in the Python programming language and demonstrated with the open-source Anser EMT system. The framework is applied in the preclinical setting though targeted liver ablation therapy on an animal model. RESULTS: The developed framework was tested with the Anser EMT electromagnetic tracking platform. Liver tumour targeting was performed using the tracking framework with the CustusX navigation platform using commercially available electromagnetically tracked needles. Ablation of two tumours was performed with a commercially available ablation system. Necropsy of the tumours indicated ablations within 5 mm of the tumours. CONCLUSIONS: An open-source framework for electromagnetic tracking was presented and effectively demonstrated in the preclinical setting. We believe that this framework provides a structure for future advancement in EMT system in and customised instrument design.

Biomechanics-based graph matching for augmented CT-CBCT.

PURPOSE: Augmenting intraoperative cone beam computed tomography (CBCT) images with preoperative computed tomography data in the context of image-guided liver therapy is proposed. The expected benefit is an improved visualization of tumor(s), vascular system and other internal structures of interest. METHOD: An automatic elastic registration based on matching of vascular trees extracted from both the preoperative and intraoperative images is presented. Although methods dedicated to nonrigid graph matching exist, they are not efficient when large intraoperative deformations of tissues occur, as is the case during the liver surgery. The contribution is an extension of the graph matching algorithm using Gaussian process regression (GPR) (Serradell et al. in IEEE Trans Pattern Anal Mach Intell 37(3):625-638, 2015): First, an improved GPR matching is introduced by imposing additional constraints during the matching when the number of hypothesis is large; like the original algorithm, this extended version does not require a manual initialization of matching. Second, a fast biomechanical model is employed to make the method capable of handling large deformations. RESULTS: The proposed automatic intraoperative augmentation is evaluated on both synthetic and real data. It is demonstrated that the algorithm is capable of handling large deformations, thus being more robust and reliable than previous approaches. Moreover, the time required to perform the elastic registration is compatible with the intraoperative navigation scenario. CONCLUSION: A biomechanics-based graph matching method, which can handle large deformations and augment intraoperative CBCT, is presented and evaluated.

Fast elastic registration of soft tissues under large deformations.

A fast and accurate fusion of intra-operative images with a pre-operative data is a key component of computer-aided interventions which aim at improving the outcomes of the intervention while reducing the patient's discomfort. In this paper, we focus on the problematic of the intra-operative navigation during abdominal surgery, which requires an accurate registration of tissues undergoing large deformations. Such a scenario occurs in the case of partial hepatectomy: to facilitate the access to the pathology, e.g. a tumor located in the posterior part of the right lobe, the surgery is performed on a patient in lateral position. Due to the change in patient's position, the resection plan based on the pre-operative CT scan acquired in the supine position must be updated to account for the deformations. We suppose that an imaging modality, such as the cone-beam CT, provides the information about the intra-operative shape of an organ, however, due to the reduced radiation dose and contrast, the actual locations of the internal structures necessary to update the planning are not available. To this end, we propose a method allowing for fast registration of the pre-operative data represented by a detailed 3D model of the liver and its internal structure and the actual configuration given by the organ surface extracted from the intra-operative image. The algorithm behind the method combines the iterative closest point technique with a biomechanical model based on a co-rotational formulation of linear elasticity which accounts for large deformations of the tissue. The performance, robustness and accuracy of the method is quantitatively assessed on a control semi-synthetic dataset with known ground truth and a real dataset composed of nine pairs of abdominal CT scans acquired in supine and flank positions. It is shown that the proposed surface-matching method is capable of reducing the target registration error evaluated of the internal structures of the organ from more than 40 mm to less then 10 mm. Moreover, the control data is used to demonstrate the compatibility of the method with intra-operative clinical scenario, while the real datasets are utilized to study the impact of parametrization on the accuracy of the method. The method is also compared to a state-of-the art intensity-based registration technique in terms of accuracy and performance.

Controlling the error on target motion through real-time mesh adaptation: Applications to deep brain stimulation.

An error-controlled mesh refinement procedure for needle insertion simulations is presented. As an example, the procedure is applied for simulations of electrode implantation for deep brain stimulation. We take into account the brain shift phenomena occurring when a craniotomy is performed. We observe that the error in the computation of the displacement and stress fields is localised around the needle tip and the needle shaft during needle insertion simulation. By suitably and adaptively refining the mesh in this region, our approach enables to control, and thus to reduce, the error whilst maintaining a coarser mesh in other parts of the domain. Through academic and practical examples we demonstrate that our adaptive approach, as compared with a uniform coarse mesh, increases the accuracy of the displacement and stress fields around the needle shaft and, while for a given accuracy, saves computational time with respect to a uniform finer mesh. This facilitates real-time simulations. The proposed methodology has direct implications in increasing the accuracy, and controlling the computational expense of the simulation of percutaneous procedures such as biopsy, brachytherapy, regional anaesthesia, or cryotherapy. Moreover, the proposed approach can be helpful in the development of robotic surgeries because the simulation taking place in the control loop of a robot needs to be accurate, and to occur in real time.

Real-Time Error Control for Surgical Simulation.

OBJECTIVE: To present the first a posteriori error-driven adaptive finite element approach for real-time simulation, and to demonstrate the method on a needle insertion problem. METHODS: We use corotational elasticity and a frictional needle/tissue interaction model. The problem is solved using finite elements within SOFA.1 For simulating soft tissue deformation, the refinement strategy relies upon a hexahedron-based finite element method, combined with a posteriori error estimation driven local -refinement. RESULTS: We control the local and global error level in the mechanical fields (e.g., displacement or stresses) during the simulation. We show the convergence of the algorithm on academic examples, and demonstrate its practical usability on a percutaneous procedure involving needle insertion in a liver. For the latter case, we compare the force-displacement curves obtained from the proposed adaptive algorithm with that obtained from a uniform refinement approach. CONCLUSIONS: Error control guarantees that a tolerable error level is not exceeded during the simulations. Local mesh refinement accelerates simulations. SIGNIFICANCE: Our work provides a first step to discriminate between discretization error and modeling error by providing a robust quantification of discretization error during simulations.

Interactive training system for interventional electrocardiology procedures.

Recent progress in cardiac catheterization and devices has allowed the development of new therapies for severe cardiac diseases like arrhythmias and heart failure. The skills required for such interventions are very challenging to learn, and are typically acquired over several years. Virtual reality simulators may reduce this burden by allowing trainees to practice such procedures without risk to patients. In this paper, we propose the first training system dedicated to cardiac electrophysiology, including pacing and ablation procedures. Our framework involves the simulation of a catheter navigation that reproduces issues intrinsic to intra-cardiac catheterization, and a graphics processing unit (GPU)-based electrophysiological model. A multithreading approach is proposed to compute both physical simulations (navigation and electrophysiology) asynchronously. With this method, we reach computational performances that account for user interactions in real-time. Based on a scenario of cardiac arrhythmia, we demonstrate the ability of the user-guided simulator to navigate inside vessels and cardiac cavities with a catheter and to reproduce an ablation procedure involving: extra-cellular potential measurements, endocardial surface reconstruction, electrophysiology mapping, radio-frequency (RF) ablation, as well as electrical stimulation. A clinical evaluation assessing the different aspects of the simulation is presented. This works is a step towards computerized medical learning curriculum.

Handling topological changes during elastic registration : Application to augmented reality in laparoscopic surgery.

PURPOSE: Locating the internal structures of an organ is a critical aspect of many surgical procedures. Minimally invasive surgery, associated with augmented reality techniques, offers the potential to visualize inner structures, allowing for improved analysis, depth perception or for supporting planning and decision systems. METHODS: Most of the current methods dealing with rigid or non-rigid augmented reality make the assumption that the topology of the organ is not modified. As surgery relies essentially on cutting and dissection of anatomical structures, such methods are limited to the early stages of the surgery. We solve this shortcoming with the introduction of a method for physics-based elastic registration using a single view from a monocular camera. Singularities caused by topological changes are detected and propagated to the preoperative model. This significantly improves the coherence between the actual laparoscopic view and the model and provides added value in terms of navigation and decision-making, e.g., by overlaying the internal structures of an organ on the laparoscopic view. RESULTS: Our real-time augmentation method is assessed on several scenarios, using synthetic objects and real organs. In all cases, the impact of our approach is demonstrated, both qualitatively and quantitatively ( http://www.open-cas.org/?q=PaulusIJCARS16 ). CONCLUSION: The presented approach tackles the challenge of localizing internal structures throughout a complete surgical procedure, even after surgical cuts. This information is crucial for surgeons to improve the outcome for their surgical procedure and avoid complications.

Robust augmented reality registration method for localization of solid organs' tumors using CT-derived virtual biomechanical model and fluorescent fiducials.

BACKGROUND: Augmented reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g., CT scan). The virtual model can be superimposed to real-time images enabling transparency visualization of internal anatomy and accurate localization of tumors. However, the 3D model is rigid and does not take into account inner structures' deformations. We present a concept of automated AR registration, while the organs undergo deformation during surgical manipulation, based on finite element modeling (FEM) coupled with optical imaging of fluorescent surface fiducials. METHODS: Two 10 × 1 mm wires (pseudo-tumors) and six 10 × 0.9 mm fluorescent fiducials were placed in ex vivo porcine kidneys (n = 10). Biomechanical FEM-based models were generated from CT scan. Kidneys were deformed and the shape changes were identified by tracking the fiducials, using a near-infrared optical system. The changes were registered automatically with the virtual model, which was deformed accordingly. Accuracy of prediction of pseudo-tumors' location was evaluated with a CT scan in the deformed status (ground truth). In vivo: fluorescent fiducials were inserted under ultrasound guidance in the kidney of one pig, followed by a CT scan. The FEM-based virtual model was superimposed on laparoscopic images by automatic registration of the fiducials. RESULTS: Biomechanical models were successfully generated and accurately superimposed on optical images. The mean measured distance between the estimated tumor by biomechanical propagation and the scanned tumor (ground truth) was 0.84 ± 0.42 mm. All fiducials were successfully placed in in vivo kidney and well visualized in near-infrared mode enabling accurate automatic registration of the virtual model on the laparoscopic images. CONCLUSIONS: Our preliminary experiments showed the potential of a biomechanical model with fluorescent fiducials to propagate the deformation of solid organs' surface to their inner structures including tumors with good accuracy and automatized robust tracking.

3D Physics-Based Registration of 2D Dynamic MRI Data.

We present a method allowing for intra-operative targeting of a specific anatomical feature. The method is based on a registration of 3D pre-operative data to 2D intra-operative images. Such registration is performed using an elastic model reconstructed from the 3D images, in combination with sliding constraints imposed via Lagrange multipliers. We register the pre-operative data, where the feature is clearly detectable, to intra-operative dynamic images where such feature is no more visible. Despite the lack of visibility on the 2D MRI images, we are able both to determine the location of the target as well as follow its displacement due to respiratory motion.